On the Commutativity of the Algebra of Invariant Differential Operators on Certain Nilpotent Homogeneous Spaces

Let G be a simply connected connected real nilpotent Lie group with Lie algebra g, H a connected closed subgroup of G with Lie algebra h and β ∈ h∗ satisfying β([h, h]) = {0}. Let χβ be the unitary character of H with differential 2 √ −1πβ at the origin. Let τ ≡ IndHχβ be the unitary representation of G induced from the character χβ of H. We consider the algebra D(G,H, β) of differential operators invariant under the action of G on the bundle with basis H\G associated to these data. We consider the question of the equivalence between the commutativity of D(G,H, β) and the finite multiplicities of τ . Corwin and Greenleaf proved that if τ is of finite multiplicities, this algebra is commutative. We show that the converse is true in many cases. 1. Notations and formulation of the question Let G be a simply connected connected real nilpotent Lie group with Lie algebra g and H a connected closed subgroup of G with Lie algebra h. For l ∈ g∗, we denote by g(l) the Lie algebra of the stabilizer G(l) of l under the co-adjoint action Ad∗ of G on g∗. For β ∈ h∗ satisfying β([h, h]) = {0}, the homomorphism β induces a character χβ of H with 2 √ −1πβ as differential at the origin. We then form the unitary induced representation τ ≡ IndHχβ of G in Hτ realized, in the usual way, as the completion of a vector subspace of C∞(G,H, β), namely the vector space of the C∞ complex functions f on G satisfying the following covariance relation: f(hg) = χβ(h)f(g) ∀h ∈ H ∀g ∈ G. (1.1) The action of G is given by right translations: τ(g′)(f)(g) = f(gg′) ∀(g; g′) ∈ G×G ∀f ∈ C∞(G,H, β). (1.2) We denote by K(G,H, τ) the subspace of C∞(G,H, β) of elements with compact support modulo H . Then the norm || ||τ on K(G,H, τ) is given by || f ||τ= ∫ H\G | f(g) | dġ (1.3) where dġ denotes a right G-invariant measure on H\G. The Hilbert space Hτ is just the completion of K(G,H, τ) relative to this norm. Moreover, the unitary Received by the editors March 17, 2000. 2000 Mathematics Subject Classification. Primary 43A85, 22E27, 22E30. c ©2001 American Mathematical Society